Pilot study of collective decision-making in the code review process

Political arguments and voting requirements often follow a simple majority method which is a decision rule that selects various alternatives that have a majority, that is, more than half the votes. On the other hand, Q&A services such as StackExchange and Yahoo! Answers use the simple majority method only as a reference point. In other words, even if an answer receives a majority vote (which represents a preferred answer for followers), the questioner might not accept the answer. In this case, the voting is used only as a reference. Open Source Software (OSS) projects might also use the voting approach only as a reference on the code review process, because a core reviewer (who makes the final decision) might not integrate a patch which was approved by reviewers into the version control system. Likewise, the votes from other reviewers might also only be used as a reference. This study identifies how many patch sets for code review follow the simple majority method in collective decision-making among reviewers. As a case study, we have analysed the criteria needed to integrate patch sets using Qt project data. From the results, we have found that only 59.5% patch sets followed the simple majority method. Patch sets with more negative votes than positive votes were likely to be rejected. Furthermore, the last vote in the first review also impacted the final decision in the first review.